Cascade classifiers may be used to implement rare object detection in images. Such a cascade classifier comprises a plurality of detection stages that have been trained to detect whether a given image is part of or contains a type of rare object, such as a face. The cascade classifier comprises a plurality of detection stages, with each detection stage deciding whether to accept the image and propagate it to a subsequent stage or to reject the image and label it as a negative. If the image reaches the final detection stage and is accepted, the image is labeled as a positive.